HDFS-15842. HDFS mover to emit metrics. by LeonGao91 · Pull Request #2738 · apache/hadoop

LeonGao91 · 2021-03-03T07:12:29Z

No description provided.

Jing9 · 2021-03-08T07:08:33Z

...hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/package-info.java

Do we need to add this file?

It is to pass the checkstyle as I added a new file to this package without package-info..

Jing9 · 2021-03-08T07:27:37Z

hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java

Does Balancer have a similar metrics? If not shall we use this PR or create a new one to do that?

We have it in the internal branch with similar implementation but it is not in upstream yet.

Actually, there has been a JIRA long time ago for balancer metrics. https://issues.apache.org/jira/browse/HDFS-10648

I can rebase our changes there as a different PR after this.

hadoop-yetus · 2021-06-17T05:30:21Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 33s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	30m 45s		trunk passed
+1 💚	compile	1m 22s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 14s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 2s		trunk passed
+1 💚	mvnsite	1m 25s		trunk passed
+1 💚	javadoc	0m 57s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 27s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 9s		trunk passed
+1 💚	shadedclient	16m 25s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 12s		the patch passed
+1 💚	compile	1m 12s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 12s		the patch passed
+1 💚	compile	1m 6s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	1m 6s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 53s		hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 100 unchanged - 1 fixed = 100 total (was 101)
+1 💚	mvnsite	1m 14s		the patch passed
+1 💚	javadoc	0m 47s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 20s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 8s		the patch passed
+1 💚	shadedclient	15m 51s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	237m 4s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 45s		The patch does not generate ASF License warnings.
		320m 52s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2738/3/artifact/out/Dockerfile
GITHUB PR	#2738
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 89ceb310f5a7 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `4dc6df6`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2738/3/testReport/
Max. process+thread count	3672 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2738/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Jing9

Thanks for updating the patch! The patch LGTM. Only one minor comment.

Jing9 · 2021-06-18T21:32:46Z

...hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/MoverMetrics.java

+
+  public static MoverMetrics create(Mover mover) {
+    MoverMetrics m = new MoverMetrics(mover);
+    DefaultMetricsSystem.instance().unregisterSource(m.getName());


Any reason we want to call unregister here? Can we call unregister at the end of the mover running?

You are right, this is not needed here. As discussed I will shutdown metrics at the end of the mover run

hadoop-yetus · 2021-06-19T05:49:28Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	13m 16s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	30m 53s		trunk passed
+1 💚	compile	1m 22s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 19s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 3s		trunk passed
+1 💚	mvnsite	1m 26s		trunk passed
+1 💚	javadoc	0m 57s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 29s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 9s		trunk passed
+1 💚	shadedclient	16m 28s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 10s		the patch passed
+1 💚	compile	1m 12s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 12s		the patch passed
+1 💚	compile	1m 10s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	1m 10s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 54s		hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 100 unchanged - 1 fixed = 100 total (was 101)
+1 💚	mvnsite	1m 16s		the patch passed
+1 💚	javadoc	0m 46s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 21s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 9s		the patch passed
+1 💚	shadedclient	15m 57s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	230m 30s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 46s		The patch does not generate ASF License warnings.
		327m 18s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestRollingUpgrade

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2738/4/artifact/out/Dockerfile
GITHUB PR	#2738
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux cab7ae036721 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `96525f6`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2738/4/testReport/
Max. process+thread count	3829 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2738/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Jing9 · 2021-06-19T22:39:14Z

+1

Jing9 reviewed Mar 8, 2021

View reviewed changes

LeonGao91 added 2 commits June 16, 2021 17:07

Add metrics to mover

abec10d

Improve style

4dc6df6

LeonGao91 force-pushed the HDFS-15842-dev branch from 26469e2 to 4dc6df6 Compare June 17, 2021 00:08

Jing9 reviewed Jun 18, 2021

View reviewed changes

Resolve comment

96525f6

Jing9 merged commit 643dfd6 into apache:trunk Jun 19, 2021

kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request Nov 24, 2021

HDFS-15842. HDFS mover to emit metrics. (apache#2738)

adc4360

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-15842. HDFS mover to emit metrics.#2738

HDFS-15842. HDFS mover to emit metrics.#2738
Jing9 merged 3 commits intoapache:trunkfrom
LeonGao91:HDFS-15842-dev

LeonGao91 commented Mar 3, 2021

Uh oh!

Jing9 Mar 8, 2021

Uh oh!

LeonGao91 Mar 8, 2021

Uh oh!

Jing9 Mar 8, 2021

Uh oh!

LeonGao91 Mar 8, 2021 •

edited

Loading

Uh oh!

hadoop-yetus commented Jun 17, 2021

Uh oh!

Jing9 left a comment

Uh oh!

Jing9 Jun 18, 2021

Uh oh!

LeonGao91 Jun 19, 2021

Uh oh!

hadoop-yetus commented Jun 19, 2021

Uh oh!

Jing9 commented Jun 19, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

LeonGao91 commented Mar 3, 2021

Uh oh!

Jing9 Mar 8, 2021

Choose a reason for hiding this comment

Uh oh!

LeonGao91 Mar 8, 2021

Choose a reason for hiding this comment

Uh oh!

Jing9 Mar 8, 2021

Choose a reason for hiding this comment

Uh oh!

LeonGao91 Mar 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Jun 17, 2021

Uh oh!

Jing9 left a comment

Choose a reason for hiding this comment

Uh oh!

Jing9 Jun 18, 2021

Choose a reason for hiding this comment

Uh oh!

LeonGao91 Jun 19, 2021

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Jun 19, 2021

Uh oh!

Jing9 commented Jun 19, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

LeonGao91 Mar 8, 2021 •

edited

Loading